Method to insert ad content into a video scene

ABSTRACT

A post-processing of video content in order to provide a capability to insert ad content, having a method to produce a modified video content from an original video content. The method comprising: analyzing the original video content to detect at least one marker, said marker comprising a marker surface delimited by a marker identification; loading profile data of a target user; determining, based at least on the profile data a replacement content; adjusting the replacement content into the marker surface; and replacing into the original video content the marker surface by the adjusted replacement content to produce the modified video content.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims priority from European Patent Application No.17180263.0 filed Jul. 7, 2017, the contents of which is herebyincorporated by reference herein in its entirety.

INTRODUCTION

The present disclosure refers to the post-processing of video content inorder to provide a capability to insert ad content.

BACKGROUND

In known solutions, a video content is edited to add one or more targetareas before being tailored to different audience. Then at a laterstage, these specific areas are replaced by alternate images appropriateto specific viewer groups or geographical regions. For example,billboards at a ground or around of a major sporting event are qualifiedto be specific areas, and these target areas are electronicallysubstituted by alternate images that are more appropriate for aparticular country or region. In particular, such a system is useful tocreate multiple television feeds each having different electronicallygenerated advertisement content which is tailored according to anintended audience.

With the advent of the individual video experience, this concept hasbeen refined up to the individual video stream. However, at theproduction side, it is simply not possible to produce a large number ofvideo feeds tailored for each user.

A known solution is to add a description file to the video feed, thisdescription file indicating the location, orientation, size of theeditable objects. This solution raises the problem of the compatibilitywith existing systems since this file should be bound with the videostream such as a proprietary sub-stream. A lot of systems along the linefrom the server to the target device are not suitable to accommodatethis type of transmission and this solution is then reserved to aconfidential audience for which the equipment was modified to acceptthis format.

SUMMARY

It is proposed a method to produce a modified video content from anoriginal video content, wherein the method comprises:

analyzing the original video content to detect at least one marker, saidmarker comprising a marker surface delimited by a marker identification,

loading profile data of a user of a client module,

determining, based at least on the profile data, a replacement content,

adjusting the replacement content into the marker surface,

replacing into the original video content the marker surface by theadjusted replacement content to produce the modified video content.

The present case is based on the recognition of a physical mark into ascene and to replace it by a selected ad content.

According to an aspect of the method, the marker identification may bein a form of a dotted line exhibiting a marker value, the analysis ofthe marker comprising the detection of the marker value from the markeridentification;

According to another aspect of the method, the determination of thereplacement content may be further based on the marker value.

According to another aspect of the method, the adjustment of thereplacement content may comprise:

-   -   determining the shape of the marker surface,    -   comparing the shape of the marker surface with the shape of the        replacement content,    -   applying a morphing function to the replacement content so as to        keep the perspective ratio while adjusting the replacement        content to the marker surface.

According to another aspect of the method, the marker value may define amarker category, said marker category being used to determine thereplacement content.

According to another aspect of the method, the modified content may beproduced in a host device, and the replacement content may be preloadedinto the host device.

According to another aspect of the method, the modified content may beproduced in a client module, and the method may comprise:

-   -   transmitting by the client module to a management center, a        request comprising at least the marker identification and the        profile data,    -   determining by the management center, based on the marker        identification and the profile data the replacement content,    -   receiving by the client module from the management center the        replacement content.

According to another aspect, the method may comprise:

-   -   receiving by the management center, together with the request, a        client identification,    -   storing in a record related to the client identification, the        marker value and a reference of the replacement content.

According to another aspect, the method may comprise:

-   -   transmitting the modified content to the client module,    -   receiving an interaction of the user in reference with the        modified content,    -   creating a snapshot of a current image of the modified content        containing the replacement content,    -   transmitting the snapshot, the user's interaction, and a client        identification to a management center,    -   detecting the replacement content from the current image and the        user's interaction,    -   recording, in a record related to the client identification, the        interaction with the replacement content.

The present disclosure also proposes a client module configured toproduce a modified video content from an original video content, saidclient module comprising,

-   -   an input module for receiving the original content,    -   a marker detector to detect at least one marker, said marker        comprising a marker surface delimited by a marker        identification,    -   a profile matcher to load user's profile data,    -   an image seeker to load, based at least on the profile data, a        replacement content,    -   a replacement module to adjust the replacement content into the        marker surface and to replace into the original video content        the marker surface by the adjusted replacement content to        produce the modified video content.

According to another aspect of the client module, the replacement modulemay be further configured to:

compare the shape of the marker surface with the shape of thereplacement content,

apply a morphing function to the replacement content so as to keep theperspective ratio while adjusting the replacement content to the markersurface.

SHORT DESCRIPTION OF THE FIGURES

The present application will be better understood thanks to the attachedfigures in which:

FIG. 1 illustrates a real environment for which an image is acquired

FIGS. 2, 3A, 3B and 4 are objects intended to carry ad content,

FIGS. 5, 6 and 7 illustrate the process of adjusting an image to theavailable frame,

FIG. 8 shows the resulting image for a give user,

FIG. 9 illustrates various examples of frames,

FIG. 10 illustrates the modules involved in the transformation of theinitial image to a modified image,

FIG. 11 illustrates one embodiment with the interaction form the user,

FIG. 12 is a block diagram illustrating components of a client module orComputing Device.

DETAILED DESCRIPTION

A first step of the method comprises identifying suitable surfaces forreceiving ad content. In FIG. 1 are shown three objects 1A, 1B and 1C aswell as a part of a wall 1D. Any surface could be used to embed adcontent. A second step comprises creating markers, a marker beingdefined by a marker identification and a marker surface, said markeridentification delimitating or surrounding the marker surface.

In FIG. 1 , four markers are thus illustrated for potentially four adcontents. The marker's identification is placed around the marker'ssurface and has a particular pattern. A plain line is not suitable sincemany plain lines can be found in a natural environment. The marker'sidentification bears a recognizable pattern once the image is acquired.

In an example of FIG. 2 , a bottle comprises a first image: a cow is forexample represented, surrounded by a particular dotted line. This lineis the marker's identification. In this example the marker's surfacealready comprises a first image. In the example 1D of FIG. 1 , themarker's surface is bare or has the background of the whole scene.

FIG. 3A shows a marker's identification surrounding a black surface. Theblack surface may be for example a mirror as illustrated in FIG. 3B, ablack board or a picture.

In FIG. 4 , the marker's identification is placed on a screen.

It is to be noted that the marker's identification can be lateridentified and discriminated from natural lines surrounding a surface.Although the dotted line is a candidate for such a function, it ispreferable that the line carries a plurality of bits of informationrepeated at least twice.

Example of dotted lines are:

A.

B.

C.

The succession of short and long portions can exhibit information suchas A: 11001100 . . . , B: 110110110 . . . or C: 10101010 . . . . Thisinformation is named marker's value.

FIG. 9 illustrates various example of the shape of the markers. A closedsurface of any shape is surrounded by the marker's identification. Thesize has no importance; a marker can be placed on an advertisementcovering a wall. Different shapes or colors of the markers can be usedto define any surface. For example, the marker could also be:

D.

FIG. 10 illustrates the processing of a video signal into one devicecarrying out the ad content replacement method. This device is named aclient module. Examples of client modules can be a smartphone or atablet and in these examples, the video signal can be produced from theinternal camera. The client module can be a rendering device such as atelevision, a VR headset, a computer. In this case, the video signal isreceived via a connection to a server.

Once the video signal is received, it is first analyzed with a MarkerDetector module. This module scans the image to detect markers. In theexample of FIG. 1 , four markers are detected. The marker'sidentification is determined and the respective marker's value iscalculated.

The marker's value defines the type of marker, such as for example theenvironment, the location, the ad category or the surface owner. Thismarker's value is sent to the profile unit in order to determine thereplacement content. The profile unit comprises the user's preferencessuch as for example previous videos watched, age group, area of living,type of preferred goods etc. The profile data as well as the marker'svalue is sent to an Image Seeker module, in charge of retrieving thereplacement content.

According to a first embodiment, the Image Seeker comprises a pluralityof images ready for replacement. The device can upload several imagesbased on the user's profile. The images are stored, for example inrespect of a category. According to an example, a category is food orcars.

This category is compared with the marker's value from which a categorycan be extracted. The stored image matching the extracted category isselected and sent to the Replacement Module.

According to a second embodiment, the Image Seeker initiates aconnection with a management center MC and communicates at least themarker's value to the management center. The Image Seeker can optionallycommunicate additional data such as the client identification, theprofile data of the user, the location of the marker, the location ofthe device, the type of device or a user's identification, etc. Themanagement center MC then determines, based on the marker's value, theappropriate replacement image. The replacement content can be an imageor a video.

The replacement content is sent to the device for processing. Forstatistic purpose, the management center can keep track of the number ofreplacement requests as well as the marker's value and the additionaldata provided by the device.

Once the replacement content is received, the Replacement Module is incharge of processing the initial image and replacing the marker surfaceby the replacement content.

FIGS. 5 to 7 illustrate an example of the processing to insert thereplacement content into the marker surface. FIG. 6 shows thereplacement content as it is received from the management center MC.FIG. 5 shows the marker identification on an object or a wall. As anexample, the television 1C of FIG. 1 is a good support for a marker.

In case that the shape of the marker identification and the replacementcontent is not the same, an adjustment is to be carried out. In anembodiment, a resize function is applied in order to adjust the markersurface with the replacement content.

In another embodiment, a morphing function as illustrated in FIG. 7 isexecuted on the replacement content so as to fit into the marker surfacewhile keeping the same perspective.

The Marker Detector communicates with the Replacement Module to informthe latter of the location of the marker surface as well as theidentification of the marker. The Replacement Module updates each imagefrom the incoming video stream as long as the Marker Module produces thelocation data of the marker. The morphing function is applied not onlyonce but each time the shape of the marker changes.

FIG. 8 illustrates a result of the Replacement Module: the Marker Modulehas produced 4 set of data, one per marker. Four replacement images havebeen received by the Image Seeker and provided to the ReplacementModule.

In the example 8A, the cow on the bottle was replaced by anotheradvertisement. In 8B, an advertisement for Zermatt replaces the blacksurface. The selection of this image is the result of the user'sprofile, which for example comprises sport and ski interest, and themarker category.

On the television screen, a replacement image with the portable phonewas inserted. A morphing function was applied on the image received asreplacement image so as to fit the shape of the marker. Concerning theelement 8D, an image of a car was inserted in the background of theinitial image at the location defined by the marker.

An application of the above method is the virtual visit of a town. Theend user connects via a rendering device to a server and can navigatealong the streets to enjoy the monuments or architectural constructions.In prior steps, some of the advertisement surfaces have been modified toadd the marker identification surrounding the ad surface. The serverembeds the modules described in reference to the FIG. 10 andautomatically updates the physical advertisement by a virtualadvertisement. The town as well as the advertisement contractorparticipating to such an experience has the possibility to adjust theadvertisement proposed to the virtual visitor. Since the lifetime of anadvertisement is short, as for a football match for example, and thevirtual tour row data are produced once, the virtual advertisement hasthe advantage to be up-to date. It can be a specific advertisement forthe virtual visitors or the currently displayed advertisement, takinginto account that each advertisement surface comprises a markerindicating a marker value.

The replacement advertisement can be also tailored for the usercurrently enjoying the virtual tour, using its preferences or language.The virtual tour provider has the possibility to sell the advertisementsurfaces individually for a given period. For that purpose, the clientmodule, while requesting the virtual visit, can also transfer otherinformation such as the client identification and the profile data.

In FIG. 11 , the modified video is prepared in the client module. Thisclient module can be a local device such as a PC or a set-top box or canbe located in a remote location such as a head-end. The original VRcontent is received by the client module and the marker is detected andreplaced by ad content, creating a modified VR content, i.e. a newscene. The modified VR content is then transmitted to the viewerrendering device. With a mouse, a joystick, a button on a VR helmet orsimply on a touch screen the viewer can interact with the new scene.Once the viewer has detected ad content, he can acknowledge the adcontent by clicking on the ad content, or anywhere on the new scene, forexample if there is only one ad. The viewer's action can be a positiveacknowledgment or a negative acknowledgment. The viewer can for exampleselect a corresponding area or button to express his acknowledgement as“|like”/“|don't like”. The viewer's device generates an image locationcorresponding to the location selected by the viewer on the new scene,and adds the client identification.

This information is sent to the management center MC in charge ofdetecting the ad content matching the received image location.

When the management center MC receives the replacement request from theImage Seeker of the host device, the request comprises the markeridentification, the user's profile and a client identification. Asalready explained, the management center MC stores this information foraccounting purpose.

When the management center MC receives a user's acknowledgementcomprising the client identification, this identification is used toretrieve the replacement content for this specific clientidentification. With the location indication, the management centerdetermines the ad content inserted into the modified video content ornew scene. With an information as “like” or “dislike”, the profile ofsaid user can be updated and the updated profile will be used later toselect the appropriate replacement content. The management center canrecord the user's interaction with the ad content and keep a creditaccount for said user. This credit can be later claimed againstreduction on products or invitation to events etc.

According to an embodiment, the client module creates a snapshot of theimage, or the new scene, transmitted to the user when the useracknowledges an ad content. The snapshot image and the image locationindicated by the user are sent to the management center. The managementcenter can then use the image location information to analyze thesnapshot to detect the ad content.

The snapshot can be stored in the management center MC as a proof of thepublication of the advertisement.

As shown is FIG. 12 , a block diagram illustrating components of aclient module or Computing Device 400, according to some exampleembodiments, is able to read instructions from a machine-readable medium(e.g., a machine-readable storage medium) and perform any one or more ofthe methodologies discussed herein. Specifically, FIG. 12 shows adiagrammatic representation of the Computing Device 400 in the exampleform of a computer system and within which instructions 403 (e.g.,software, a program, an application, an applet, an app, or otherexecutable code) for causing the Computing Device 400 to perform any oneor more of the methodologies discussed herein may be executed. Inalternative embodiments, the Computing Device 400 operates as astandalone device or may be connected (e.g., networked) to othermachines. In a networked deployment, the Computing Device 400 mayoperate in the capacity of a server machine or a client machine in aserver-client network environment, or as a peer machine in apeer-to-peer (or distributed) network environment. The Computing Device400 may be a server computer, a client computer, a personal computer(PC), a network router, a network switch, a network bridge, or anymachine capable of executing the instructions 403, sequentially orotherwise, that specify actions to be taken by that machine. Further,while only a single machine is illustrated, the term “machine” shallalso be taken to include a collection of machines that individually orjointly execute the instructions 403 to perform any one or more of themethodologies discussed herein.

The Computing Device 400 includes a processor 402 (e.g., a centralprocessing unit (CPU), a digital signal processor (DSP), an applicationspecific integrated circuit (ASIC), or any suitable combinationthereof), a main memory 404, and a static memory 406, which areconfigured to communicate with each other via a bus 408. The processor402 can be a single chip or a multi-core processor (Core 1 . . . Core4). The Computing Device 400 may further include a Network Interface 410in communication with other external devices.

The Computing Device 400 includes a first storage unit I 414 on which isstored the instructions 403 embodying any one or more of themethodologies or functions described herein. The instructions 403 mayalso reside, completely or at least partially, within the main memory404, within the processor 402 (e.g., within the processor's cachememory), or both, during execution thereof by the Computing Device 400.Accordingly, the main memory 404 and the processor 402 may be consideredas machine-readable media. The Computing Device 400 further contain asecond storage unit II 415 on which is stored the User's Data 416. Oneexample of the User's Data is the user's identification UserID and theprofile data used to select the replacement content.

The Computing Device 400 comprises various modules to produce themodified video content from the original video content. The originalvideo content is received by the video input V In (or acquired through acamera) and processed by the Computing device. As explained above, theComputing Device comprises a Marker detector module 417, a ProfileModule 418, a Image Seeker module 419 and an Replacement Module 419 forproducing the modified video content V Out.

As used herein, the term “memory” refers to a machine-readable mediumable to store data temporarily or permanently and may be taken toinclude, but not be limited to, random-access memory (RAM), read-onlymemory (ROM), buffer memory, flash memory, and cache memory. While thestorage unit 414, 415 is shown in an example embodiment to be a singlemedium, the term “storage unit” should be taken to include a singlemedium or multiple media (e.g., a centralized or distributed database,or associated caches and servers) able to store instructions.

The instructions 403 may further be transmitted or received over acommunications network 409 or 411 using a transmission medium via thenetwork interface device 410 and utilizing any one of a number ofwell-known transfer protocols (e.g., HTTP). Examples of communicationnetworks include a local area network (LAN), a wide area network (WAN),the Internet, mobile telephone networks, POTS networks, and wirelessdata networks (e.g., WiFi and WiMAX networks). The term “transmissionmedium” shall be taken to include any intangible medium that is capableof storing, encoding, or carrying instructions for execution by themachine 400, and includes digital or analog communications signals orother intangible medium to facilitate communication of such software.

Throughout this specification, plural instances may implementcomponents, operations, or structures described as a single instance.Although individual operations of one or more methods are illustratedand described as separate operations, one or more of the individualoperations may be performed concurrently, and nothing requires that theoperations be performed in the order illustrated. Structures andfunctionality presented as separate components in example configurationsmay be implemented as a combined structure or component. Similarly,structures and functionality presented as a single component may beimplemented as separate components. These and other variations,modifications, additions, and improvements fall within the scope of thesubject matter herein.

The invention claimed is:
 1. A computing device to produce a modifiedvideo content from an original video content, said computing devicecomprising: at least one memory for storing the original video content;at least one processor; a marker detector module configured to detect,from the original video content, at least one marker, said markercomprising a marker surface surrounded by a marker identification, andto determine a marker value from the marker identification; a managementcenter configured to receive a request for the modified video contentfrom a user of a client module, and to determine, based on the requestand the marker value, a replacement content; and a replacement moduleconfigured to replace the marker surface of the original video contentwith the replacement content to produce the modified video content. 2.The computing device of claim 1, wherein the computing device is aserver computer.
 3. The computing device of claim 1, wherein thereplacement module is further configured to adjust the replacementcontent into the marker surface.
 4. The computing device of claim 3,wherein the replacement module is further configured to: determine ashape of the marker surface; compare the shape of the marker surfacewith a shape of the replacement content; and apply a morphing functionto the replacement content to keep a perspective ratio while adjustingthe replacement content to correspond to the marker surface.
 5. Thecomputing device of claim 1, wherein the marker identification exhibitsa binary marker value that is decodable.
 6. The computing device ofclaim 1, wherein the marker identification is in a form of a dotted lineexhibiting a marker value.
 7. The computing device of claim 1, whereinthe request comprises a user's profile.
 8. The computing device of claim1, wherein the computing device is further configured to: transmit themodified content to a client module; receive a snapshot of a currentimage of the modified content containing the replacement content fromthe client module; receive an interaction of the user in reference withthe modified content from the client module; transmit the snapshot, theuser's interaction, and a client identification to the managementcenter; detect the replacement content from the current image and theuser's interaction; and record, in a record related to the clientidentification, the interaction with the replacement content.
 9. Thecomputing device of claim 1, wherein the marker value defines a markercategory, said marker category being used to determine the replacementcontent.
 10. The computing device of claim 1, wherein the managementcenter is further configured to: receive, together with the request, aclient identification; and store in a record related to the clientidentification, the marker value and a reference of the replacementcontent.